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6 TECHNICAL FIELD 

7 This invention relates generally to document browsers and pages that are displayed 

8 using such browsers. More particularly, the invention provides a method and apparatus for 

9 allowing a user to add annotations and other markings to an electronic document 

10 independently of the content of the document. 

11 BACKGROUND OF THE INVENTION 

12 Web browsers are conventionally used to display documents created with a standard 

13 descriptive language such as Hyper Text Markup Language (HTML). Documents written in 

14 such languages can be displayed with various visual and audio effects such as color, 

15 animation, sound effects, and the like. The documents may include text, bitmap images, 

16 graphics, and other types of content. 

17 Web browsers (or, more generally, "document browsers") typically provide functions 

1 8 which permit a user to advance forward and backward among a series of related pages; to jump 

19 to a specified page; to jump to pages that are referenced via a so-called "hyperlink" embedded 

20 in a page; and to maintain a list of recently viewed pages. Although frequently used to 

2 1 navigate through web pages on the Internet, document browsers can be used to display pages 

22 that reside on a local area network or even a local computer on which the browser executes. 

23 Recently, as computers have become smaller, it has become possible to run browsers 

24 on notebook computers, palm-sized computers, and so-called "tablet and stylus" computers. 

25 The latter devices typically comprise a thin computer roughly the size of a sheet of paper with 

26 a large writing surface that doubles as a display, and a stylus that substitutes for a keyboard. 

27 A user can display and edit documents using the stylus as a writing instrument. 

28 One potential application for such hand-held computers is the display of mass- 

29 produced documents such as books, maps, and other descriptive material. For example, a user 

30 who purchases a textbook or fictional work on a CD-ROM or other medium can display the 

3 1 work using a small computer such as a tablet-based computer. 
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1 In some instances, a user may want to annotate or otherwise mark pages displayed on 

2 the computer. Using a conventional word processing program, a user can rearrange text, 

3 highlight certain words or phrases, or delete portions of the text. Such features require that the 

4 user have access to the underlying content (e.g., the descriptive material that forms the work). 

5 In the case of copyrighted materials, however, editing capabilities may not be available to 

6 modify the underlying materials. Certain text may be copy-protected or otherwise unavailable 

7 for editing using normal editing tools, or may not be in a format suitable for word processing. 

8 In such circumstances, there may be no practical way for a user to annotate such works. As 

9 one example, a student viewing a copy-or edit-protected textbook on a tablet-based computer 
10 may have no practical way of making notes directly in the textbook. Such a feature would be 



1 1 desirable notwithstanding the copy protection or unavailability of general editing functions for 

12 the document. 

13 SUMMARY OF THE INVENTION 

14 The present invention provides a method and apparatus for users to annotate 



1 5 documents in a document browser without changing the contents of the documents. According 

16 to one variation of the invention, a tablet and stylus-based computer is progranraied with a 

17 document browser that permits a user to annotate documents that are viewed through the 

18 browser. The annotations are stored separately from the viewed document pages but are 

19 correlated with the pages such that when a previously annotated page is revisited, annotations 

20 relating to that page are retrieved and displayed on top of the page as an "ink" layer. 

21 According to one variation of the invention, three different annotation modes are 

22 provided: ink, highlight, and erase. Each mode can be selected through a user interface 

23 supplied through the browser or through controls embedded in a document. The annotation 

24 functions can be implemented as an Active-X® control, as an Applet, or as dedicated code 

25 integrated into the browser. Other features and advantages of the invention will become 

26 apparent through the following description, the figures, and the appended claims. 

27 BRIEF DESCRIPTION OF THE DRAWINGS 



28 FIG. 1 shows a conventional general-purpose computing environment that can be 

29 employed in various embodiments of the invention. 

30 FIG. 2 shows a conventional tablet and stylus-based computer that can be used 
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1 according to one aspect of the invention. 

2 FIG. 3 is a conceptual view showing an ink layer 302 arranged on top of an underlying 

3 document page 301. The ink layer may include opaque and translucent markings. 

4 FIG. 4 shows a user interface on a computer screen including a menu 402 that permits 

5 a user to select from among a plurality of annotation modes on a displayed document. 

6 FIG. 5 shows a user interface on a computer screen including an opaque annotation 

7 501 and a highlighted annotation 502, both superimposed over a displayed document. 

8 FIG. 6 shows a user interface on a computer screen including an opaque aimotation 

9 601 in a pre-erase state. 

10 FIG. 7 shows a user interface on a computer screen including an opaque annotation 

11 701 in a post-erase state, where the annotation has been partially erased. 

12 FIG. 8 shows one possible software design for implementing various principles of the 

13 invention. 

14 FIG. 9 A shows a flowchart indicating various steps that can be executed to provide an 

1 5 annotation function according to certain inventive principles. 

16 FIG. 9B shows a flowchart indicating how previously created annotations can be 

1 7 superimposed over a document page. 

18 DETAILED DESCRIPTION OF THE INVENTION 

19 FIG. 1 is a schematic diagram of a conventional general-purpose digital computing 

20 environment that can be used to implement various aspects of the invention. Computer 100 



21 includes a processing unit 110, a system memory 120, and a system bus 130 that couples 

22 various system components including the system memory to the processing unit 110. The 

23 system bus 130 may be any of several types of bus structures including a memory bus or 

24 memory controller, a peripheral bus, and a local bus using any of a variety of bus architectures. 

25 The system memory includes read only memory (ROM) 140 and random access memory 

26 (RAM) 150. 

27 A basic input/output system 160 (BIOS), containing the basic routines that help to 

28 transfer information between elements within the computer 100, such as during start-up, is 

29 stored in ROM 140. Computer 100 also includes a hard disk drive 170 for reading from and 

30 writing to a hard disk (not shown), a magnetic disk drive 1 80 for reading from or writing to 
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1 a removable magnetic disk 190, and an optical disk drive 191 for reading from or writing to 

2 a removable optical disk 192 such as a CD ROM or other optical media. The hard disk drive 

3 1 70, magnetic disk drive 180, and optical disk drive 191 are connected to the system bus 130 

4 by a hard disk drive interface 192, a magnetic disk drive interface 193, and an optical disk 

5 drive interface 194, respectively. The drives and their associated computer-readable media 

6 provide nonvolatile storage of computer readable instructions, data structures, program 

7 modules and other data for the personal computer 100. It will be appreciated by those skilled 

8 in the art that other types of computer readable media which can store data that is accessible 

9 by a computer, such as magnetic cassettes, flash memory cards, digital video disks, Bernoulli 

10 cartridges, random access memories (RAMs), read only memories (ROMs), and the like, may 

1 1 also be used in the exemplary operating environment. 

12 A number of program modules can be stored on the hard disk, magnetic disk 190, 

13 optical disk 192, ROM 140 or RAM 150, including an operating system 195, one or more 

14 application programs 196, other program modules 197, and program data 198. A user can enter 

15 commands and information into the computer 100 through input devices such as a keyboard 

16 101 and pointing device 102. Other input devices (not shown) may include a microphone, 

1 7 joystick, game pad, satellite dish, scanner, or the like. These and other input devices are often 

1 8 connected to the processing unit 110 through a serial port interface 106 that is coupled to the 

19 system bus, but may be connected by other interfaces, such as a parallel port, game port or a 

20 universal serial bus (USB). A monitor 107 or other type of display device is also connected 

21 to the system bus 130 via an interface, such as a video adapter 108. In addition to the monitor, 

22 personal computers typically include other peripheral output devices (not shown), such as 

23 speakers and printers. 

24 The computer 100 can operate in a networked environment using logical connections 

25 to one or more remote computers, such as a remote computer 109. Remote computer 109 can 

26 be a server, a router, a network PC, a peer device or other common network node, and 

27 typically includes many or all of the elements described above relative to computer 100, 

28 although only a memory storage device 111 has been illustrated in FIG. 1. The logical 

29 cormections depicted in FIG. 1 include a local area network (LAN) 112 and a wide area 

30 network (WAN) 113. Such networking environments are commonplace in offices, enterprise- 
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1 wide computer networks, intranets and the Internet. 

2 When used in a LAN networking environment, the computer 100 is connected to the 

3 local network 112 through a network interface or adapter 114. When used in a WAN 

4 networking environment, the personal computer 100 typically includes a modem 1 1 5 or other 

5 means for establishing a communications over the wide area network 113, such as the Internet. 

6 The modem 115, which may be internal or extemal, is connected to the system bus 1 30 via the 

7 serial port interface 106. In a networked environment, program modules depicted relative to 

8 the personal computer 100, or portions thereof, may be stored in the remote memory storage 

9 device. 

10 It will be appreciated that the network connections shovm are exemplary and other 

1 1 means of establishing a communications link between the computers can be used. The 
^ 1 2 existence of any of various well-known protocols such as TCP/IP, Ethernet, FTP, HTTP and 
O 1 3 the like is presumed, and the system can be operated in a client-server configuration to permit 
m 14 a user to retrieve web pages from a web-based server. Any of various conventional web 
^ 1 5 browsers can be used to display and manipulate data on web pages. 

□ 1 6 FIG. 2 shows a tablet and stylus computer that can be used in accordance with various 

r 17 aspects of the present invention. Any or all of the features, subsystems, and fiinctions in the 

J 1 8 system of FIG. 1 can be included in the computer of FIG. 2. Computer 201 includes a large 

P 19 display surface 202 (e.g., a flat panel display) on which a plurahty of windows 203 is 

J 20 displayed. Using stylus 204, a user can select, highlight, and write on the display area. 

- 2 1 Computer 20 1 interprets marks made using stylus 204 in order to manipulate data, enter text, 

22 and execute conventional computer application tasks such as spreadsheets, word processing 

23 programs, and the like. One commercially available tablet and stylus computer incorporating 

24 many of these features is the Stylistic 2300 computer sold by Fujitsu Personal Systems, Inc., 

25 of Santa Clara, CaHfomia. 

26 A stylus could be equipped with buttons or other features to augment its selection 

27 capabilities. In one embodiment, a stylus could be implemented as a "penciF' or *'pen" in 

28 which one end constitutes a writing portion and the other end constitutes an "eraser" end 

29 which, when moved across the display, indicates that portions of the display are to be erased. 

30 Other types of input devices such as a mouse, trackball, or the like could be used. 
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1 Additionally, a user's own finger could be used to select or indicate portions of the displayed 

2 image on a touch-sensitive or proximity-sensitive display. Consequently, the term "user input 

3 device" is intended to have a broad definition and encompasses many variations on well- 

4 known input devices. 

5 FIG, 3 is a conceptual view showing a computer screen on which a document page 301 

6 is displayed using a document browser. As is conventional, the document page contains text 

7 and/or graphics of interest to the user (i.e., "content"). In accordance with the invention, a 

8 special "ink" layer 302 is superimposed over the document page 301. Using stylus 305, the 

9 user can highlight certain parts of the document 304, which are then shown in a highlighted 

10 color (e.g., yellow) against the background text. 

1 1 Also using stylus 305, the user can mark opaque annotations 303 on the computer 

12 display, which appear to be vmtten on the page itself. (Although ink layer 302 is shown as 

1 3 a separate layer in FIG. 3, this view is for conceptual purposes only; the user would see only 

14 document page 301 with aimotations superimposed over the text and/or graphics on the page). 

1 5 According to one aspect of the invention, annotations made by a user on document page 301 

16 are maintained as a separate "ink" layer 302 that is superimposed over and blended with pixels 

1 7 on the document page. Consequently, even though the annotations appear to be made directly 

1 8 on the underlying document, they are actually maintained in a separate display layer blended 

1 9 with the document page being viewed. 

20 In one embodiment, annotations are specific to each user, such that other users who 

21 view the same document will not see a previous user's annotations. This feature can be 

22 implemented using log-ins or other user identifiers. For an identical document retrieved at two 

23 different remote computers each having the inventive features, annotations made on one 

24 remote computer would not show up on the other remote computer, since the annotations are 

25 made independently of the document content. It is, however, within the scope of the invention 

26 to permit sharing of annotations if desired. 

27 Although the inventive principles are described with respect to a tablet and stylus 

28 computer, these principles can be applied to other types of computers, including desktop 

29 systems, notebook computers, and the like. Many types of input devices, such as touch- 

30 sensitive displays and the Uke, can be used to indicate those portions of the display that are to 
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be annotated. Moreover, as explained above, many variations on a basic stylus are of course 
possible, including an embodiment in which the stylus includes a writing end and an "erasing" 
end. 

FIG. 4 shows a user interface on a computer screen on which a document page 401 is 
displayed using a document browser. In accordance with one embodiment of the invention, 
the user interface includes an annotation mode selection menu 402 that permits a user to select 
from among a plurality of annotation modes for a displayed document. By tapping the stylus 
on pen button 403, the user can cause the annotation mode selection menu 402 to be displayed. 
A mode indicator 404 indicates the current annotation mode for the document browser. 

According to one variation of the invention, these modes can include an "ink" mode, 
a "highlight" mode, and an "erase" mode. Additionally, a "selection" mode permits the user 
to switch to normal browser functions. As shovra in FIG. 4, the user interface is already in 
"erase" mode (indicated by mode indicator 404), so that "erase" is not shown as one of the 
available modes in menu 402. 

It is not necessary to provide an explicit "mode selection" feature as described above; 
instead, mode selection can be implicit in the manner by which the input device is used. As 
one example, a stylus could be used to provide an ink mode when the stylus is used as a 
writing implement; a highlighting mode when the stylus is held a certain way or when a button 
is pressed on the stylus; and an erase mode when an "erasing" end of the stylus is moved 
across portions of the display. As another example, one of the modes (e.g., the ink mode) 
could be established as a default mode when a user input device is manipulated, with other 
modes being selected based on additional user inputs or manipulations or even voice 
commands, for example. Many other variations are of course possible. 

FIG. 5 shows a user interface on a computer screen including the document of FIG. 4. 
Using the "ink" mode, the user has annotated a portion of the document 501 with an opaque 
ink color (e.g., black or blue). The user has also highlighted a different portion of the 
document 502 using the "highlight" mode. The highlighted portion 502 appears in a 
translucent color (e.g., yellow) superimposed over the underlying document. Using the "ink" 
and "highlight" modes, the user is able to annotate pages viewed through the document 
browser as needed. 
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1 FIGS. 6 and 7 show one possible approach for implementing an "erase" mode. FIG. 

2 6 shows a pre-erase screen in which the user has annotated a document page using an opaque 

3 ink color 601 . FIG. 7 shows the same screen of FIG. 6 after the user has selected the "erase" 

4 mode and moved the stylus across a portion of the annotated region 601, leaving a partially 

5 erased annotation 701. Using the "erase" mode, the user can remove previous annotations 

6 from the computer screen. The width of the eraser function can be varied to suit various 

7 applications, or it can even be user-selectable. 

8 FIG. 8 shows one possible software design for implementing various principles of the 

9 invention. As shown in FIG. 8, a document browser 801 permits a user to retrieve and view 

10 content in the form of documents from a document storage area 804. The documents can be 

1 1 stored locally on the user's computer (e.g., on CD-ROM disks and the like), or they can be 

12 stored remotely and retrieved across a network such as the Internet. Aimotation editor 803 

1 3 retrieves and stores annotations in "ink layers" stored in a storage device or area 805. As with 

14 document storage, ink layer storage area 805 can be local or remotely located. 

15 Aimotations retrieved from storage area 805 are provided to pixel blending function 

16 802, which is indicated as a newly provided function within document browser 801. 

17 Alternatively, pixel blending function 802 can be provided separately (e.g., in the operating 

1 8 system, for example). In general, pixel blending function 802 blends pixels from a document 

19 page with corresponding pixels from an annotation or "ink" layer mapped to that document 

20 page, and generates a blended pixel image that is displayed as an annotated document page 

21 806. 

22 In one embodiment, each opaque annotation pixel replaces a corresponding pixel on 

23 a document page, whereas each translucent annotation pixel (e.g., highlighting) is blended 

24 with a corresponding pixel value on a document page to produce a different color from that 

25 in the original document. Pixel blending functions such as alpha blending are well known and 

26 no further elaboration of this feature is required. 

27 Annotations can be saved as "strokes" in a data structure and stored locally in a file 

28 associated with the currently displayed page of the document. Each stroke can comprise a 

29 stroke type (e.g., annotate, highlight or erase), a stroke width, and a stroke color in addition 

30 to coordinates indicating the stroke trajectory. In various embodiments, annotation storage 
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1 occurs automatically, such that whenever the input device is released (or lifted), a new stroke 

2 is automatically stored. 

3 Alternatively, it is of course possible to store annotations as a bitmap image having the 

4 same or similar dimensions as the presently displayed document page, such that the annotation 

5 image can be superimposed over the document page. Combinations of these approaches are 

6 of course possible. For example, opaque annotations could be stored in a data structure, while 

7 highlighting could be stored as a bitmap image. 

8 According to one embodiment, document browser 801 generates a title change event 

9 to annotation editor 803 each time a different document page is selected through document 

10 browser 801. This permits annotation editor 803 to retrieve previously stored annotations for 

1 1 a page, and to create a new annotation file for a newly displayed page. Annotation editor 803 

1 2 intercepts stylus movements fi-om a stylus or other cursor movement device and provides them 

13 to document browser 801 after any annotation processing has been performed. In one 

14 embodiment, browser 801 can constantly generate "screen paint" commands that are 

1 5 intercepted so that a blended image is continuously generated. 

16 Annotation editor 803 can be implemented as an Active-X® control element, as an 

17 Applet, or as other forms of software fiinctions. Annotation editor 803 and any related 

1 8 fiinctions can be embedded into a document viewed through browser 801, such that viewing 

19 the document provides the annotation fiinctions. 

20 FIG. 9A shows a flowchart indicating various steps that can be executed to provide an 

21 annotation fiinction according to one embodiment. It will be appreciated that many different 

22 approaches for carrying out the inventive principles are possible, and the steps shown are 

23 intended to be exemplary only. For example, although the steps are illustrated as being 

24 performed sequentially, they can in fact be performed by different components at different 

25 times in an object-oriented system using event-driven processing techniques. The steps shown 

26 can be implemented in software that is stored on a storage medium such as a disk. 

27 Beginning in step 901, a stylus "down" event is detected. As noted previously, any of 

28 various input devices such as a mouse with key clicks or others can be used. In step 902, a test 

29 is performed to determine whether the annotation mode has been enabled. (See FIG. 4). If 

30 the annotation mode is not presently active, then in step 903 processing retums to the browser 
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1 (i.e., the stylus movement command is passed through to document browser 801). 

2 In step 904, after it has been determined that the annotation mode is active, the page 

3 identifier for the currently displayed page is retrieved. It is contemplated but not necessary 

4 that each document page have a unique identifier (e.g., a unique URL). In step 905, a test is 

5 performed to determine v^hether the currently displayed page has a previously existing "ink" 

6 or "annotation" file associated therewith. If such an ink file exists, it is retrieved from storage 

7 area 805 (see FIG. 8). If no such file exists, a new ink file is created in step 906, preferably 

8 using a name that readily corresponds to the displayed document page. 

9 After an ink file has been retrieved or newly created, a test is performed in step 907 to 

1 0 determine whether the annotation mode is "ink" mode. If the current annotation mode is "ink" 

1 1 mode, then in step 91 1 opaque pixels are vmtten into the ink file at the current stylus position. 

12 The number of pixels vmtten can be varied depending on the width or "swath" to be used for 

13 the annotation. For a wide swath (e.g., a magic marker type of annotation), a plurality of 

14 pixels can be written surrounding the current stylus position. For a narrower swath (e.g., a 

15 pinpoint type of annotation), a smaller number of pixels (e.g., one pixel wide) can be written 

16 to the annotation file at the current stylus position. It is within the scope of the invention to 

17 permit the user to select or change the width of the annotation marks. 

18 In step 914, a pixel blending operation is performed, such that the pixels in the ink file 

19 are blended with pixels from the document, which causes the docvunent to be displayed with 

20 annotations. (As described above, opaque pixels may entirely replace those in the viewed 

21 document). In step 915, a test is performed to determine whether the stylus or other input 

22 device has been lifted or disengaged. If so, then in step 916 the ink layer is stored, and 

23 processing returns to the browser in step 917. Otherwise, annotations continue at step 902, 

24 and can be stored as part of the same stroke. If the user again presses the stylus or input device 

25 against the document, then annotations resume in step 901 . 

26 Returning to step 907, if the present mode is not "ink" mode, then a test is made in 

27 step 908 to determine whether highlight mode was selected. If highlight mode is active, then 

28 in step 912 translucent pixels are written at the current stylus position. For example, yellow, 

29 pink, or other translucent colors can be stored at the current stylus position. As before, the 

30 pixels are blended in step 914, changing the resulting pixels on output but nevertheless 
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1 permitting the user to see the original document pixels in a modified form. Processing 

2 continues as described previously. 

3 In step 908, if highlight mode is not currently active, then in step 909 a test is 

4 performed to determine whether erase mode is active. If so, then in step 91 3 the pixels at the 

5 current stylus position are cleared or zeroed out. As with highlight mode, the width of the 

6 erasure can be preset to a swath wider than one pixel, and the width can be different than the 

7 highlight or annotation pixel width. As before, the remaining non-erased pixels are blended 

8 with the original document image in step 914 and processing continues until the ink layer is 

9 stored. 

10 FIG. 9B shows a flowchart indicating how previously created annotations can be 

1 1 superimposed over a document page. In step 918, a title change event is detected in document 
^ 12 browser 801, and is sent to annotation editor 803, In various embodiments, a title change 
£= 13 event occurs whenever the presently viewed document page changes. This may occur when 
jh 14 the user jumps to a new document page, or when the user moves forward or backward among 
^ 15 a sequence of related document pages, such as might occur in a book or a set of related 
D 16 hyperlinks. Other types of events, such as a "status change" event, for example, can be used 

: 53 

1 ^ 3 

; ' 17 instead of a title change event. 

^ 18 {)j5C7In step 91 9, a tept is performed to determine whether an ink layer already exists for the 

□ 19 new document page./If no ink layer exists for the page, processing retums to the document 

.J' 20 browser and the page is displayed normally. If an ink layer exists for the page, then in step 

S 21 920 the ink laygns retrieved. In step 921, the pixels from the retrieved ink layer are blended 

22 with the displ^ed page and the resulting image appears on the computer display. Thereafter, 

23 processingretums to the browser. A separate ink layer file can be created for each document 

24 page, or one ink file can be created for an entire document and indexed according to pages on 

25 wluch the annotations appear. Other approaches are of course possible. 

26 Thus has been described a system, method, and apparatus for annotating electronic 

27 document pages using an annotation editor that stores annotations independently of the content 

28 of the underlying document. It will be appreciated that many modifications and variations of 

29 the invention are possible, and the scope of the invention is not limited by the specific 

30 examples and descriptions herein. 
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